Exploring the diversity and invariance in yourself for visual pre-training task

Recently, self-supervised learning methods have achieved remarkable success in the visual pre-training task. By simply pulling different augmented views of each image together or other novel mechanisms, they can learn much unsupervised knowledge and significantly improve transfer performance models. However, these works still cannot avoid representation collapse problem, i.e., only focus on limited regions extracted features totally inside are nearly same. Generally, this problem makes models sufficiently describe multi-grained information images, which further limits upper bound their performance. To alleviate issue, paper introduces a simple but effective mechanism, called Exploring Diversity Invariance Yourself (E-DIY). pushing most view away, E-DIY preserve diversity region-level features. similar from same together, ensure robustness Benefiting above invariance exploring better extracts compared with previous approaches. Extensive experiments various downstream tasks demonstrated superiority our method, e.g., there is 1.9% improvement (compared recent state-of-the-art BYOL) AP50 metric detection task VOC.